… env.step

Rollout timing is currently reported only as a total per trajectory
(`generate/trajectory_completion_time_*`, added in NovaSky-AI#1804). That total cannot
distinguish an engine-bound rollout from an environment-bound one, which is the
first question to ask when GPU utilization sags during generation.

Accumulate, per trajectory in `agent_loop`, the wall-clock time spent awaiting
`inference_engine_client.generate()` and the time spent in `env.step()`, and
report:

  generate/trajectory_llm_time_{mean,p90,max}
  generate/trajectory_env_time_{mean,p90,max}
  generate/frac_time_in_env

`frac_time_in_env` is time-weighted (sum over sum) so long trajectories count
proportionally rather than each trajectory contributing equally.

The two components need not sum to `e2e_time`; the remainder is tokenization,
chat-template rendering and other in-loop bookkeeping. Both are plumbed like the
existing `e2e_time`: optional, and omitted entirely if any trajectory in the
batch did not record them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Item 1's llm/env time split was computed only at the per-generate()
get_rollout_metrics call. Every logging path (async trainer, eval,
step-wise) re-aggregates through concatenate_generator_outputs, which had
no raw llm/env lists and fell back to the substring-based extra_keys
aggregator. There p90 and frac_time_in_env hit the sum() branch, so a
16-group mini-batch logged frac_time_in_env near 12 and p90 near 16x its
real value.

Store the raw per-trajectory llm/env lists on GeneratorOutput like
e2e_time, thread them through concatenate_generator_outputs with the same
step-wise last-step filter, and recompute all seven metrics there so the
extra_keys fallback leaves them alone.

Also in this commit:
- Extend test_llm_vs_env_time_split_metrics to assert the split on a
  concatenated output, the path that actually broke.
- Fix the phase_metrics module docstring query. The gauge has a lowercase
  `phase` tag and no shared label with the GPU metric, so
  `avg(ray_node_gpus_utilization) by (Phase)` does not run; replace it
  with a vector match against `ray_skyrl_training_phase{phase="eval"}`.
- Deduplicate the last-step filter and the frac-block array construction.

Review feedback pass:
- Add trajectory_llm_times/env_times to the GeneratorOutput field
  guardrail test, which this branch had broken.
- Extract _add_time_stats for the mean/p90/max blocks (completion, llm,
  env) and reuse its sums for frac_time_in_env.
- Extract _optional_times for the omit-if-any-None per-prompt lists.
- Use _concat_field for stop_reasons and rollout_logprobs too.
- Cut duplicated comments; the llm/env split was explained in six
  places, now documented once per surface.
- Assert the step-wise per-step replication of the split in the
  existing step-wise timing test.

Review feedback: the llm and env times traveled as parallel named fields
(llm_time/env_time on two dataclasses, trajectory_llm_times/
trajectory_env_times on GeneratorOutput), so every hop re-documented and
re-plumbed each field, and the omit-if-any-None handling needed its own
helper per field.

One time_splits dict per trajectory (keys "llm", "env") and one
trajectory_time_splits dict-of-lists on GeneratorOutput replace them.
The split semantics are documented once, on TrajectoryOutput.time_splits.
_optional_times is gone; get_rollout_metrics loops the dict through
_add_time_stats generically, so a future component is a new key rather
than a new field at every layer. Metric names are unchanged.

_concat_time_splits keyed off the first output, so concatenating a batch
that recorded splits with one that did not crashed when the missing
batch came second. Fold list and dict-of-lists handling into one
_concat_optional_field with any-missing-is-None semantics for every
optional field, and let _last_step_only recurse into dicts. Adds an
order-independence regression test.